One aspect of computer security involves protecting computer systems from malicious software, also known as “malware.” Malware comes in many forms; however, many common varieties of malware perform writes or other accesses to unauthorized locations in computer memory. For example, certain malware modifies critical in-memory system data structures to gain control over a computer. One such attack involves overwriting the system call table so that code supplied by the malware is executed in place of one or more system calls. Such an attack may be used by so-called “rootkits” to gain control of a system and evade detection. As another example, some malware causes the system to execute code from unauthorized memory segments such as the system stack or data segments. For example, buffer overflow exploits, return-oriented programming (ROP) gadgets, and similar exploits cause the system to execute code from (or driven by) the system stack or heap that should not ordinarily be executed. Malware may perform such attacks to execute so-called “shellcode” attacks; that is, to execute arbitrary code (usually introduced from a remote location) on the computer system.
Typical computer security systems attempt to detect malware by applying code analysis to classify potentially malicious code. For example, a computer security system may perform static analysis of code to search for well-known malware signatures. Some systems may also perform dynamic analysis of code execution. Typical dynamic monitoring such as single-stepping execution using a hypervisor, has a large negative impact on system performance.
Some computer processors provide support for hardware transactional memory. Transactional memory allows the programmer to designate a segment of code called a “transaction” to execute independently and atomically. That is, memory operations occurring within the transaction are not visible to other transactions or threads executing on the computing system until the transaction is successfully committed. After successful commit, all memory changes made during the transaction are instantaneously available to other threads on the system. Transactional memory may be implemented by speculatively executing the transaction, detecting any memory conflicts that occur during execution of the transaction, and then aborting and rolling back the transaction in response to memory conflicts. Memory conflicts include, for example, a transaction attempting to write to a memory location that has already been read or written-to by another transaction. Transactional memory may simplify the programming model for parallel computing. One commercially available example of hardware transactional memory support is Transactional Synchronization Extensions (Intel® TSX), available on certain processors manufactured by Intel® Corporation.